feat: updated loaders #14594

florian-lefebvre · 2025-10-21T11:43:14Z

Todo

Changes

Closes Deprecate loader types generation from schema #14381
What does this change?
Be short and concise. Bullet points can help!
Before/after screenshots can help as well.
Don't forget a changeset! Run pnpm changeset.
See https://contribute.docs.astro.build/docs-for-code-changes/changesets/ for more info on writing changesets.

Testing

Docs

changeset-bot · 2025-10-21T11:43:19Z

⚠️ No Changeset found

Latest commit: 771939a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

florian-lefebvre · 2025-10-28T13:00:00Z

@ascorbic this is very WIP but can you have a look at the code and let me know what you think about the approach?

ascorbic

Can you clarify the scope of what you are trying to fix here? As I understand it, there are a couple of issues that could be fixed by running the schema parse later:

resolve references
support refine etc with image helper (and generally improve the image handling)
don't require loader authors to manually parse data

Is the goal to fix both of these, or soemthing else?

packages/astro/src/content/content-layer.ts

ascorbic · 2025-11-04T10:23:18Z

packages/astro/src/content/mutable-data-store.ts

+		new Traverse(data).forEach((_, val) => {
+			if (typeof val === 'string' && val.startsWith(IMAGE_IMPORT_PREFIX)) {
+				const src = val.replace(IMAGE_IMPORT_PREFIX, '');
+				foundAssets.add(src);
+			}
+		});


If we're doing this two-pass approach, could we find a better way to handle images? This has always felt like a hack, but was needed because of the time when it had to run

Probably, I chose the simplest approach possible for this POC by simply copying

I looked into this and I don't think the new approach would allow to get rid of this. Basically during the first pass we don't know which properties are images so we can't store them for processing before the 2nd pass IIUC

florian-lefebvre · 2025-11-04T14:14:59Z

So the goal here is to support returning a schema from load() instead of having it as a top level property. AFAIK this means:

parseData() cannot, well, parse data anymore because the schema may not be known already. We only get the final schema after running load()
set() / get() etc must still work on unparsed data

So my solution to this problem was to have an intermediary store. It contains the initial parsed data and the loader can manipulate it by calling any method. Then once load() is done, we replay events on the real data store.

All the potential improvements you mentionned are basically unintended and out of scope of this PR (except parseData())

ascorbic · 2025-11-04T14:28:06Z

It seems to me that if we're going to fix this, we should at the very least ensure this is designed this in such a way that the others can be fixed too, even if we don't fix them immediately

florian-lefebvre · 2025-11-04T16:25:34Z

Sure. Can you give me more info about the first 2 bullet points? Like how it works currently and why, and what improvement you'd like

ascorbic · 2025-11-04T17:05:03Z

We currently can't check references because other collections may not have been loaded. If doing two passes, we can check that referenced entries exist. This should be a relatively easy win.

With images, because of the way we do this placeholder thing and then process images later, we can't return useful metadata from the helper anymore, meaning refine() doesn't work. I'm not sure if this is possible to fix with this, because it may still be too early in the build, but it's worth looking at.

florian-lefebvre · 2025-11-05T14:12:36Z

@ascorbic I pushed a POC for early references validation, let me know what you think

packages/astro/src/content/content-layer.ts

ascorbic · 2025-11-06T10:51:15Z

packages/astro/src/content/content-layer.ts

+								new Traverse(data).forEach((ctx, value) => {
+									if (!isReference(value)) {
+										return;
+									}
+									// TODO: better data structure
+									const collectionStore = filteredRawLoaderResults.find(
+										(e) => e.name === value.collection,
+									)!.memoryStore;
+									const id = 'id' in value ? value.id : value.slug;
+									if (!collectionStore.has(id)) {
+										// TODO: AstroError
+										throw new Error(
+											`Invalid reference for ${name}.${ctx.keys!.map((key) => key.toString()).join('.')}: cannot find entry ${id}`,
+										);
+									}
+								});


Is there no way to do this inside the schema validation? Traversing always seems a bit of a hack

Yeah I agree. If reference was passed like image() we could have some context in there but AFAIK with the current implementation there's not much we can do

Oh, good point. I forgot it was handled like that. Now I'm trying to remember why it didn't need to traverse before

I think that's because reference() is actually never checked? It would only throw at runtime when calling eg. getEntry(post.data.author)

I think pre-content layer it did check

Ah yeah definitely, I didn't understand that's what you were talking about

Okay so I looked and that's because createReference was passed the lookupMap

astro/packages/astro/src/content/runtime.ts

Line 607 in 32b14f2

const entry = store.get(collection, lookup);

florian-lefebvre · 2025-11-07T12:48:55Z

I think the POC works fine so now I'll work on cleaning everything up, tests, docs etc

wip

c227582

florian-lefebvre added this to the v6.0.0 milestone Oct 21, 2025

florian-lefebvre self-assigned this Oct 21, 2025

github-actions bot added the pkg: astro Related to the core `astro` package (scope) label Oct 21, 2025

florian-lefebvre added 2 commits October 28, 2025 13:16

Merge branch 'next' into feat/updated-loaders

e79202f

wip

17d6570

feat: simplify

2b4e38e

florian-lefebvre linked an issue Oct 28, 2025 that may be closed by this pull request

Deprecate loader types generation from schema #14381

Open

ascorbic reviewed Nov 4, 2025

View reviewed changes

florian-lefebvre added 4 commits November 5, 2025 14:31

Merge branch 'next' into feat/updated-loaders

e350966

feat: simplify types

75d6661

feat: update name

532143b

feat: early references check poc

68a810f

florian-lefebvre added 6 commits November 5, 2025 15:17

chore: clean

cec45ce

chore: clean

3d36f09

chore: clean

05d1954

chore: clean

f03e5f2

fix: test

ad5a7f6

Merge branch 'next' into feat/updated-loaders

6bac68f

florian-lefebvre requested a review from ascorbic November 6, 2025 10:37

florian-lefebvre commented Nov 6, 2025

View reviewed changes

packages/astro/src/content/content-layer.ts Show resolved Hide resolved

ascorbic reviewed Nov 6, 2025

View reviewed changes

florian-lefebvre mentioned this pull request Nov 7, 2025

chore(content-layer): clean #14726

Open

florian-lefebvre added 3 commits November 7, 2025 14:02

feat: split

cb08d45

feat: improve

c8567da

chore: todo

771939a

Uh oh!

feat: updated loaders #14594

Are you sure you want to change the base?

feat: updated loaders #14594

Conversation

florian-lefebvre commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Todo

Changes

Testing

Docs

Uh oh!

changeset-bot bot commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

florian-lefebvre commented Oct 28, 2025

Uh oh!

ascorbic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

florian-lefebvre commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ascorbic commented Nov 4, 2025

Uh oh!

florian-lefebvre commented Nov 4, 2025

Uh oh!

ascorbic commented Nov 4, 2025

Uh oh!

florian-lefebvre commented Nov 5, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

florian-lefebvre commented Nov 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

florian-lefebvre commented Oct 21, 2025 •

edited

Loading

changeset-bot bot commented Oct 21, 2025 •

edited

Loading

florian-lefebvre commented Nov 4, 2025 •

edited

Loading